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PREFACE 


This  effort  was  conducted  under  exploratory  development  Work  Unit 
7184  10  44,  "Advanced  Strategic  Cockpit  Engineering  and 
Research,"  by  personnel  of  the  Crew  Station  Integration  Branch, 
Human  Engineering  Division,  Crew  Systems  Directorate,  of  the 
Armstrong  Laboratory,  Wright-Patterson  Air  Force  Base,  Ohio.  The 
Sensor  Fusion  Flight  Demonstration  (SFFD)  project,  which  is  the 
broader  context  within  which  the  reported  research  was  conducted, 
was  sponsored  by  the  Theater  Missile  Defense  Office  of  the 
Aeronautical  Systems  Center,  Wright-Patterson  Air  Force  Base, 

Ohio  (ASC/YXT) .  Lt  Col  D.  Bostelman  (ASC/YXT)  served  as  the 
management  focal  point  and  Mr  W.  Moyer  (ASC/YXT)  provided 
technical  direction  for  the  SFFD  project. 

Special  thanks  are  due  to  the  highly  professional  and 
knowledgeable  personnel  of  the  Strategic  Air  Command  (SAC)  who 
served  as  subject  matter  experts  in  support  of  the  SFFD  project. 
Additional  thanks  are  due  to  Maj  B.  J.  Aller  (HQ  SAC)  for  freely 
contributing  his  expertise  in  the  area  of  flight  test  planning, 
to  Capt  J.  Sato  (HQ  SAC)  for  her  guidance  in  requirements  and 
tactics  to  hold  critical  mobile  targets  at  risk,  and  to  Maj  A. 
Sobel  for  her  assistance  in  performing  SFFD  crew  task 
decompositions . 


TABLE  OF  CONTENTS 

SECTION 

TITLE 

PAGE 

PREFACE 

iii 

LIST  OF  FIGURES 

vi 

GLOSSARY 

vii 

I 

INTRODUCTION 

1 

BACKGROUND 

1 

HUMAN  FACTORS  ISSUES 

3 

II 

SYSTEM  CONCEPT 

5 

SFFD  SUBSYSTEMS 

5 

TACTICS 

14 

III 

OPERATOR  INTERFACE 

17 

SFFD  CREW  STATION 

17 

CREW  TASKS 

17 

IV 

SUBJECT  MATTER  EXPERTS  (SMEs) 

33 

SMEs 

33 

GROUND  SCHOOL 

34 

REFRESHER  TRAINING 

35 

V 

MEASURES 

36 

BASELINE 

36 

SFFD  01  ENHANCEMENT  CONCEPTS 

38 

OPERATIONAL  UTILITY  ASSESSMENT 

39 

OPTIMIZING  THE  SFFD  01 

41 

VI 

RESULTS 

42 

WORKLOAD 

42 

PERFORMANCE 

42 

iv 

TABLE  OF  CONTENTS  (Continued) 


SECTION  TITLE  PAGE 

SFFD  01  ASSESSMENT  43 
OPERATIONAL  UTILITY  ASSESSMENT  46 
SFFD  01  ENHANCEMENTS  46 
SFFD  01  OPTIMIZATION  47 


VII  CONCLUSIONS  48 

WORKLOAD  48 

PERFORMANCE  48 

SFFD  01  49 

OPERATIONAL  UTILITY  49 

ENHANCEMENTS  49 

OPTIMIZATION  50 

REFERENCES  51 


DTIC  QUALITY  INSPECTED  3 


V 


LIST  OF  FIGURES 


FIGURE  TITLE  PAGE 

1  SFFD  System  Concept  6 

2  SFFD  Sensor  Management  System  12 

3  Notional  Sequence  of  SFFD  Events  15 

4  SFFD  Tactics  16 

5  SFFD  Operator  Interface  18 

6  PTP  Controls  for  Select  Format 

and  Select  Screen  Functions  20 

7  SFFD  Sensor  Plan  Format  21 

8  Situational  Awareness  Display  Format  24 

9  FLIR  and  LLS  Freeze  Frame  Image 

Display  Screens  25 

10  PTP  Controls  for  FLIR  and  Laser  Line 

Scan  Freeze  Frame  Control  26 

11  SFFD  Target  Data  Format  30 

12  Notional  Operational  SFFD  Scenarios  40 

13a  Results  of  Rating  Scale  for  Baseline 

SFFD  Operator  Interface  Design 

Attributes  44 

13b  Results  of  Rating  Scale  for  Baseline 

SFFD  Operator  Interface  Design 

Attributes  (Continued)  45 


GLOSSARY 


AGL 

ASD 

ATC 

ATR 

DIR 

DMA 

DTED 

E 

FAR 

FLIR 

FOV 

FRZ 

ft 

G 

HR 

HVP 

Hz 

INS 

IP 

IRIG 

GPS 

km 

Kts 

LADAR 


Above  ground  level 
Aeronautical  Systems  Division 
Automatic  target  cuer 
Automatic  target  recognizer 
Downward-looking  infrared 
Defense  Mapping  Agency 
Digital  terrain  elevation  data 
Mental  effort  (SWAT) 

False  alarm  rate 

Forward-looking  infrared 

Field-of-view 

Freeze  frame  display  mode 

Feet 

Giga 

Hit  rate 

High  value  point 
Hertz 

Inertial  navigation  system 
Initial  point 

Interrange  instrumentation  group 
Global  positioning  system 
Kilometer 
Knots 

Laser  radar 


vii 


LLS 

LOC 

m 

MMWR 

MOP 

MPD 

mr 

N 

Nm 

01 

Pixel 

Pro-SWAT 

PTP 

RCS 

S 

SA 

SAC 

SFFD 

SME 

SMS 

SWAT 

T 

TDC 

TEL 

TH 

W 


Laser  linescan 

Line  of  communication 

Meter 

Millimeter  wavelength  radar 
Measure  of  performance 
Multipurpose  display 
Milliradian 
Narrow 

Nautical  mile 
Operator  interface 
Picture  element 
Projective  SWAT 
Programmable  touch  panel 
Radar  cross  section 
Psychological  stress  (SWAT) 

Situational  awareness 

Strategic  Air  Command 

Sensor  fusion  flight  demonstration 

Subject  matter  expert 

Sensor  management  subsystem 

Subjective  workload  assessment  technique 

Time  stress  (SWAT) 

Target  designation  cursor 
Transporter/erector/ launcher 
Track  handle 
Wide 


via 


WPT 

Waypoint 

o 

Degree 

/im 

Micrometer 

/X 


Section  I 


INTRODUCTION 


BACKGROUND 

The  Sensor  Fusion  Flight  Demonstration  (SFFD)  project  was 
initiated  in  December  1990  under  the  sponsorship  of  the  Mobile 
Target  Office  of  the  Aeronautical  Systems  Division  (ASD) ,  Wright- 
Patterson  Air  Force  Base,  Ohio.  The  overall  intent  of  the 
project  was  to  demonstrate  an  increased  Air  Force  capability  to 
locate  concealed  and/or  camouflaged  targets  (Stephens,  1992) . 

The  SFFD  project  had  two  specific  objectives: 

1.  The  fusion  of  data  from  two  or  more  sensors  in 
determining  if  a  potential  target  site  was,  in 
fact,  occupied,  and 

2.  Real-time  route  planning  integrated  with  the 
multiple  sensor  system. 

An  associated  sub-objective  was  to  assess  the  capability  of  the 
aircrew  member  in  exploiting  the  SFFD  system  concept  in  the 
execution  of  a  series  of  concealed  target  search  tactics. 

Prior  to  the  initiation  of  the  SFFD  project,  ASD  had 
sponsored  several  single  sensor  flight  demonstration  activities 
as  part  of  the  GLITTER  PAGEANT  program.  These  sensors  included 
synthetic  aperture  radar,  millimeter  wavelength  radar  (MMWR) , 
forward-  and  downward-looking  infrared  (FLIR,  DIR) ,  and  laser 
linescanners  (LLS)  and  radar  (LADAR) .  Imagery  from  the  GLITTER 
PAGEANT  collections  was  used  to  assess  the  potential  utility  of 
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each  sensor  class  and  to  support  the  development  and 
demonstration  of  automatic  target  cueing  and  recognition 
(ATC/ATR)  algorithms  and  systems.  Building  on  these  single 
sensor  demonstrations,  the  SFFD  project  was  initiated  to  allow 
the  Air  Force  to  explore  the  potential  utility  of  integrated, 
multiple  sensors.  It  was  intended  to  serve  as  an  intermediate 
concept  leading  to  the  future  flight  demonstration  of  a  fully 
developed  sensor  fusion  concept  (Toms  and  Kuperman,  1991) .  This 
"building  block"  approach  (single  sensors,  followed  by  multiple 
sensors,  leading  to  sensor  fusion)  was  viewed  as  a  way  of 
conducting  an  advanced  development  program  which  would  minimize 
the  cost,  schedule,  and  technical  risks  associated  with  the 
future  engineering  manufacturing  development  of  an  operational 
system  concept  (Peio,  Crawford,  and  Kuperman,  1991) . 

The  SFFD  project  was  conducted  by  the  Boeing  Defense  and 
Space  Group,  Seattle,  Washington.  A  Boeing  757-200  aircraft  was 
specially  modified  to  serve  as  the  SFFD  platform.  It  was 
equipped  to  support  a  variety  of  advanced  avionics  effectiveness 
demonstrations,  several  of  which  were  combined  to  form  the  SFFD 
avionics  concept.  All  SFFD  flight  activity  took  place  at  the  Ft. 
Lewis  Washington  Army  National  Guard  training  ranges  located  in 
the  Ranier  Military  Operations  Area  (South  of  Olympia, 

Washington) . 

A  series  of  six  Air  Force-sponsored  SFFD  flights  were 
conducted  during  the  period  of  22  April  through  7  May  1992.  The 
targets  of  interest  were  M-1  Abrams  main  battle  tanks  and 
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surrogates  for  theater  missile  transporter/erector/ launchers 
(TELs) .  Actual  tanks,  positioned  by  the  Washington  Army  National 
Guard,  were  either  uncovered  or  masked  with  camouflage  nets.  The 
"TELs"  were  large  transport  vehicles  (garbage  trucks)  and  were 
similarly  treated.  In  addition,  unoccupied  camouflage  nets  were 
also  deployed.  They  were  intended  to  serve  as  decoys.  Each  of 
the  six  flights  consisted  of  approximately  18  passes  through  the 
target  area.  All  passes  were  conducted  at  a  nominal  altitude  of 
1,200  ft  AGL  and  at  a  ground  speed  of  approximately  205  kts 
indicated  airspeed. 


HUMAN  FACTORS  ISSUES 

The  SFFD  project  represented  a  unique  opportunity  to  gain 
insight  into  two  areas  related  to  the  design  and  evaluation  of 
the  operator  interface  (01)  for  a  crew-aided,  multiple  sensor, 
target  acquisition  system.  First  were  those  issues  related  to 
the  design  of  the  01  itself.  These  were  divided  into  four 
sequential  areas: 

1.  Providing  situational  awareness  (SA)  information  to  the 
crewmember  prior  to  the  target  search  and  confirmation  segments 
of  the  flight. 

2.  Providing  SA  and  mission  pacing  information  to  the 
crewmember  during  the  target  search  and  confirmation  segments. 

3.  Providing  a  decision  support  capability  during  the 
target  confirmation  segment. 

4.  Providing  a  decision  support  capability  during  the 
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weapon  assignment  segment. 

The  second,  related  area  of  human  factors  concern  was  the  utility 
of  the  01  in  supporting  the  crewmember  during  the  actual 
prosecution  of  the  mission. 

The  major  objectives  were  embodied  in  several  very  specific 
human  factors  engineering  contributions  to  the  SFFD  project. 


4 


Section  II 


SYSTEM  CONCEPT 


SEED  SUBSYSTEMS 

The  SEED  avionics  suite  includes  a  MMWR  with  an  associated 
ATC,  a  ELIR,  and  a  LLS,  together  with  an  on-board  Sensor 
Management  Subsystem  (SMS) .  Eigure  1  presents  the  basic  avionics 
concept  embodied  in  the  SEED  project. 

Mission  Planning:  Ground-based  mission  planning  is  performed  to 
limit  the  area  to  be  searched  for  potential  targets.  Lakes, 
swamps,  and  other  non-traf f icable  terrain  is  excluded  from  the 
search  area.  The  remaining  area  is  subdivided  and  prioritized 
with  regard  to  search  value  on  the  basis  of  accessibility  to 
lines  of  communications  (LOCs,  i.  e.,  roads  or  trails),  degree  of 
terrain  slope,  availability  of  vegatative  masking  (e.  g., 
treelines) ,  etc.  Eurther,  mission  planning  takes  into  account 
the  type  of  search  to  be  performed,  the  sensor (s)  to  be  employed, 
and  the  direction (s)  in  which  they  are  to  be  pointed  in  order  to 
achieve  an  unobscured  line  of  sight  to  each  potential  target 
site.  The  output  of  the  mission  planning  process  is  a  data  base 
of  single  or  multiple  sites  (i.  e.,  "high  value  points,"  HVPs)  to 
be  visited  by  the  SEED  aircraft  and  sensors. 

Navigation  and  Elidht  Management:  The  testbed  aircraft  avionics 
complement  includes  both  an  Inertial  Navigation  System  (INS)  and 
a  Global  Positioning  System  (GPS)  airborne  terminal.  The 
combined  INS/GPS  navigation  reference  serves  as  a  highly  accurate 
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Figure  1.  SFFD  System  Concept 
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determinant  of  aircraft  position  and  supports  the  SFFD  SMS  in 
obtaining  sensor  coverage  of  preselected  points.  The  INS/GPS 
system  is  coupled  into  the  757 's  autopilot  to  provide  automated 
horizontal  steering  commands.  The  pilot  maintained  manual 
control  of  aircraft  flight  altitude. 

MMWR:  The  MMWR  was  developed  by  the  Boeing  High  Technology 

Center  (Henderson,  1990) .  It  serves  to  provide  a  low  altitude, 
adverse  weather  search  capability.  It  is  a  35  GHz  freguency, 
real  beam  mode  radar.  It  is  operated  as  a  frequency-modulated, 
continuous  wave  radar.  It  is  reported  to  have  an  effective  range 
from  in  excess  of  10  nm  down  to  100  meters,  with  two  km  being  the 
effective  range  for  the  SFFD  flights.  It  exhibits  a  beam  width 
(one-way)  of  2.4°  and  a  range  resolution  of  0.5  m  out  to  2  km. 

The  MMWR  scans  a  swath  of  ±  30°  directly  in  front  of  the  SFFD 
aircraft. 

ATC;  The  signal  from  the  MMWR  is  fed  directly  into  an  ATC.  The 
ATC  algorithms  (developed  during  prior  reseach  and  development 
programs)  were  refined  to  correspond  to  the  target  set  and 
backgrounds  of  interest  to  the  SFFD  project.  The  radar  cross 
section  (RCS)  of  the  tank  was  estimated  to  be  about  50  m^  while 
the  RCS  of  the  surrogate  TEL  was  approximately  three  times  as 
great.  The  MMWR  ATC  was  used  to  detect  possible  targets  (i.  e., 
objects  exhibiting  an  RCS  significantly  greater  than  that  arising 
from  background  clutter  sources) . 

FLIR;  A  FLIR  Systems  Incorporated  Model  2000B  FLIR  sensor  was 
employed  as  one  of  the  two  target  identification  sensors 
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demonstrated  in  support  of  the  SFFD  project.  This  sensor  employs 
a  two  by  four  element  time-delay-integration  array  of  mercury- 
cadium-telluride  detectors  to  detect  long  wave  length  infrared 
energy  in  the  eight  to  12  nm  spect^-al  band.  It  is  a  dual  field 
of  view  (FOV)  sensor.  The  wide  (W)  FOV  is  28°  horizontal  by  15° 
vertical,  while  the  narrow  (N)  FOV  (achieved  using  5.6 
magnification  telescope  lens),  is  5°  horizontal  by  2.4°  vertical. 
The  NFOV  was  employed  from  its  maximum  slant  range  down  to  a 
minimum  slant  range  of  2500  ft  at  which  point  the  FLIR  switched 
to  WFOV.  Resolution  of  the  FLIR  is  reported  to  be  2.7  mr  in  the 
WFOV  mode  and  0.25  mr  in  the  NFOV.  The  FLIR  images  320  active 
lines  per  frame  which  are  converted  into  a  conventional  525  TVL 
per  frame  video  display  format.  The  FLIR  is  gimbal-mounted  and 
its  FOV  may  be  directed  in  both  azimuth  (±  60®)  and  elevation  to 
any  point  on  the  ground,  under  the  command  of  the  SMS. 

The  expected  image  quality  of  the  FLIR  was  a  factor  which 
limited  the  SFFD  flight  altitude.  The  FLIR  was  considered  to  be 
more  suitable  for  security  surveillance  purposes  than  for 
tactical  target  acquisition.  However,  it  was  available  to  the 
SFFD  project  while  higher  quality  FLIRs  were  not.  The  FLIR 
gimbals  also  proved  to  be  a  limitation  on  SFFD  flight  profiles. 
Although  rated  to  a  much  higher  value,  the  gimbals  demonstrated  a 
maximum  airspeed  limitation  of  205  kts;  this  was  the  airspeed 
adopted  for  all  SFFD  flights. 

The  FLIR  images  were  presented  to  the  operator  in  a  freeze 
frame  display  format.  The  ground  locations  for  the  FLIR  images 
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were  determined  either  by  a)  "intelligence"  estimates  of  HVPs  (i. 
e. ,  likely  locations  for  the  deployment  of  the  prebriefed  target 
types,  based  on  capability  and  tactical  doctrine  assumptions)  or 
b)  MMWR/ATC  target  detections  for  which  the  SMS  was  able  to 
schedule  a  FLIR  imaging  event. 

LLS ;  The  LLS,  produced  by  Hughes-Danbury  Optical  Systems 
(formerly  Perkin-Elmer  Corporation) ,  was  employed  as  a  second 
SFFD  identification  sensor.  It  employs  a  gallium  arsenide  laser 
(8.5  ^m  wavelegth)  together  with  a  silicon  avalanche  photodiode 
detector  to  acquire  passive  (emittance) ,  active  (specular 
reflectance) ,  and/or  height  (range)  images  over  a  ±60°  swath 
centered  on  the  aircraft  nadir  line.  This  provides  approximately 
±  1,500  ft  of  cross  track  ground  coverage  when  the  aircraft  is  at 
an  altitude  of  1,000  ft  AGL. 

In  essence,  in  the  LLS  active  mode,  the  spot  on  the  ground 
being  illuminated  by  the  laser  beam  is  simultaneously  imaged  by 
the  detector.  The  LLS  has  a  reported  resolution  of  20  mr  in  the 
cross  track  direction  by  5  mr  along  track.  Because  it  is 
essentially  downward-looking,  the  LLS  offers  some  measure  of 
robustness  against  terrain-masked  ground  targets.  The  LLS  was 
expected  to  be  useful  against  camouflaged  and  concealed  targets 
because  of  the  three  dimensional  nature  of  the  range  image  and 
the  possibility  of  penetrating  signature  denial  netting. 

The  LLS  images  were  presented  to  the  operator  in  a  freeze 
frame  display  format.  The  LLS  digital  format  is  2048  picture 
elements  (pixel)  across  track  (i.  e.,  in  the  scan  direction)  by 


9 


however  long  the  along  track  path  might  be.  A  conventional, 
real-time  display  of  line  scan  sensor  imagery  is  frequently  in 
the  form  of  a  "waterfall''  format.  LLS  forward  coverage  is 
provided  by  the  forward  motion  of  the  aircraft.  The  scan  rate  is 
controlled  so  that  each  individual  scan  line  is  just  contiguous 
with  the  preceding  line.  (This  results  in  avoiding  both 
"holidays"  and  oversampling  in  the  ground  coverage.)  In  a 
"waterfall"  format,  the  newest  scan  line  is  presented  at  the  top 
of  the  display,  with  prior  lines  shifted  toward  the  bottom.  In 
the  SFFD  implementation,  however,  the  SMS  controlled  an  image 
extraction  step.  A  video  frame  grabber  captured  a  2048  by  2048 
pixel  full  area  image  from  the  "live"  LLS  video.  The  SMS 
determined  the  location  of  a  sub-image  area,  corresponding  to  512 
by  512  pixels,  which  was  centered  on  the  target  location.  This 
l/16th  area  sub-image  was  presented  in  freeze  frame  in  a 
conventional  525  TVL  video  format. 

The  ground  locations  for  the  LLS  sub-images  were  determined 
either  by  a)  "intelligence"  estimates  of  HVPs  (i.  e.,  likely 
locations  for  the  deployment  of  the  prebriefed  target  types, 
based  on  capability  and  tactical  doctrine  assumptions)  or  b) 
MMWR/ATC  target  detections  for  which  the  SMS  was  able  to  schedule 
a  LLS  imaging  event.  LLS  images,  presented  to  the  operator,  were 
within  ±47°  of  perpendicular  to  the  nadir  line. 

The  power  of  the  LLS  laser  limited  the  effective  range  of 
the  SFFD  flights  and  resulted  in  the  nominal  altitude  of  1200  ft 
AGL.  A  more  powerful  laser  would  have  supported  higher  flight 


10 


altitudes.  Again,  the  LLS  was  available  to  the  SFFD  project. 

SMS ;  The  SMS,  described  by  Wilber  (1991)  and  Lammers  (1992)  and 
depicted  in  Figure  2,  provided  multiple  capabilities  to  the  SFFD 
system.  Once  ground-based  mission  planning  has  produced  a  set  of 
prioritized  HVPs,  the  SMS  logic  was  used  to  generate  a  sensor 
management  plan.  The  SMS  included  what  was,  in  essence,  an 
expert  system  to  control  the  FLIR.  This  process  was  intended  to 
maximize  the  number  of  HVPs  actually  imaged  by  the  FLIR  and  to 
also  optimize  the  guality  of  the  imagery  obtained.  The  first 
stage  of  the  mission  planner  established  a  straight-line  aircraft 
ground  track  whick  brought  the  aircraft  into  proximity  to  the 
maximum  number  of  HVPs.  The  computations  required  to  assure 
imaging  the  maximum  number  of  HVPs  were  dominated  by  the  slew 
rate  capability  of  the  FLIR  gimbals.  The  aircraft  position, 
current  FLIR  pointing  direction,  range  to  the  HVP,  desired  sensor 
FOV,  the  number  of  images  desired  at  the  HVP,  the  location  of  any 
already  scheduled  FLIR  images  subsequent  to  the  current  HVP, 
etc. ,  were  included  in  determining  if  and  when  a  HVP  would  be 
imaged  by  the  FLIR.  In  many  cases,  the  azimuthal  range  over 
which  a  HVP  could  be  imaged  was  restricted.  This  was  also  taken 
into  consideration.  Basically,  the  sensor  plan  was  built  on  the 
timing  required  to  slew  the  FLIR  between  successive  HVPs. 

Planning  for  LLS  coverage  was  much  simpler.  The  HVP  was  either 
within  the  LLS  FOV  or  it  wasn't.  If  it  was,  then  a  LLS  image  was 
scheduled  for  display  to  the  operator. 

During  the  SFFD  flights,  the  SMS  provided  active  control 
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Figure  2.  SFFD  Sensor  Management  System 
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over  FLIR  pointing  and  LLS  image  display  to  the  operator. 
Following  the  established  sensor  plan,  the  SMS  directed  the  FLIR 
to  each  HVP  location.  When  the  MMWR/ATC  provided  a  real-time 
detection,  the  sensor  planner  function  of  the  SMS  performed  a 
real-time  rescheduling  of  FLIR  pointing.  Once  a  detection  was 
received  by  the  SMS,  it  first  attempted  to  correlate  the  location 
of  the  detection  with  an  existing  HVP  location  in  the  sensor 
plan.  If  the  detection  location  was  already  scheduled  for  sensor 
coverage,  the  SMS  did  nothing  further.  If  the  detection  was  at  a 
new  location,  then  the  SMS  attempted  to  schedule  sensor  coverage 
of  that  location.  Priority  was  given  to  the  preplanned  HVPs, 
based  on  the  a  priori  "intelligence"  estimates,  over  the  MMWR/ATC 
declarations.  Again,  sensor  gimbal  rates  determined  whether  the 
SMS  could  incorporate  the  new  image  request  into  the  existing 
sensor  plan  without  deleting  other  preplanned  sensor  coverage. 
This  capability  of  being  able  to  automatically  integrate  MMWR 
detections  as  real-time  additions  to  the  sensor  plan  allowed  the 
SFFD  to  demonstrate  dynamic  retasking.  (The  origin  of  the 
requests  for  additional  sensor  coverage  were  essentially 
transparent  to  the  SMS,  They  could,  as  easily,  have  originated 
from  any  of  a  number  of  on-board  or  off-board  cueing  or  detection 
sensors . ) 

The  SMS  also  served  to  support  two  other  SFFD  functions. 
During  the  target  confirmation  phase,  the  SMS  managed  the  data 
base  of  images  the  operator  judged  to  contain  possible  targets. 
During  the  weapon  allocation  phase,  the  SMS  managed  the  image 


13 


data  base  and  supported  the  operator  in  performing  a  target 
prioritization/weapon  assignment  function. 


TACTICS 

Figure  3  depicts  a  notional  SFFD  mission  phase.  It  is 
composed  of  three  types  of  activity:  prior  to  the  initial  point 
(pre-IP) ,  search,  and  post-search.  The  multiple  sensors  and  SMS 
support  a  variety  of  search  tactics.  Figure  4  depicts  these 
tactics.  Mission  planning  has  established  the  aircraft  track  as 
generally  parallel  to  a  LOC.  The  MMWR  can  cover  the  entire 
stretch  of  road  included  in  this  search  area,  as  can  the  LLS. 

(If  the  MMWR/ATC  results  in  a  detection,  then  the  FLIR  could  be 
slewed  by  the  SMS  to  acquire  a  confirmation  image.)  Multiple 
FLIR  images  can  be  planned  and  commanded  to  accomplish  search 
along  tree  lines  or  over  local  areas.  Individual  HVPs  can,  of 
course,  be  imaged.  Because  of  the  accuracy  of  the  SFFD 
navigation  system,  FLIR  images  can  be  commanded  to  look  into  road 
cuts,  fire  breaks,  and  other  interruptions  in  tree  lines. 
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PRE-PLANNED  HVPs 


END  MISSION  PHASE 


Figure  3.  Notional  Sequence  of  SFFD  Events 


Figure  4.  SFFD  Tactics 


Section  III 


OPERATOR  INTERFACE 


SFFD  CREW  STATION 

The  SFFD  crew  station  (the  01)  consisted  of  three 
multipurpose  displays  (MPDs)  as  information  sources  and  four 
programmable  touch  panels  (PTPs)  as  control  input  devices  (shown 
in  Figure  5) .  The  LLS  imagery  was  always  presented  on  MPD  1  and 
was  always  in  freeze  frame  mode  (FRZ) .  MPD  1  was  controlled  by 
PTP  1.  The  FLIR  imagery  was  always  presented  on  MPD  2  and  was 
always  FRZ.  MPD  2  was  controlled  by  PTP  2.  MPD  3  was  used  for 
several  purposes.  During  the  pre-IP  segment,  MPD  3  was 
configured  to  present  a  depiction  of  the  planned  route  and 
preplanned  imaging  events,  i.  e.,  HVPs  (similar  to  Figure  3). 

There  is  also  a  track  handle  (TH)  located  at  the  crew 
station.  It  is  used  to  select  a  "display  of  interest"  from 
between  the  three  MPDs,  to  control  the  position  of  a  target 
designation  cursor  (TDC) ,  and  as  a  "hands  on"  controller  for 
selecting/deselecting  individual  items  of  information.  (The 
specifics  of  the  TH  control  functions  are  presented  in  the 
context  of  the  individual  crew  tasks  which  require  them.) 


CREW  TASKS 

Pre-IP:  During  the  pre-IP  phase,  the  operator  verified  the 

configuration  of  his  crew  station,  assuring  himself  that  the 
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required  information  sources  were  presented  on  the  appropriate 
display  surfaces.  (The  use  of  a  "school  solution"  crew  station 
configuration  was  intended  to  avoid  variation  in  crew  performance 
due  to  differences  in  information  availability.)  He  also 
reviewed  a  hardcopy  "enroute  strike  folder"  (an  annotated  line 
drawing  of  the  flight  track  from  pre-IP  through  End  Mission 
Phase.  The  hardcopy  stike  map  was  annotated  to  show  local 
airfields  and  towns.  (These  were  "gamed"  as  "keep  out"  or 
"threat  avoidance"  areas  in  the  scenario.)  Range/bearing 
information  was  provided  alongside  each  flight  leg  and  at  each 
start  turn  point. 

The  operator  could  also  review  the  sensor  management  plan 
during  the  pre-IP  leg.  He  could  note  how  many  images  to  expect 
and  in  what  order  and  with  which  sensor  combination  to  expect 
them.  This  information  was  presented  on  the  Sensor  Plan  display 
format  which  was  an  alternative  information  display  on  MPD  3 
(lower  MPD) .  The  Sensor  Plan  format  was  accessed  (on  MPD  3) 
through  the  use  of  PTP  4 .  Figure  6  depicts  the  PTP  formats 
employed  to  invoke  the  Sensor  Plan.  First,  the  operator  would 
depress  the  SENSOR  DATA  button  on  the  Select  Format  PTP  screen. 
This  action  invokes  the  Select  Screen  menu  on  the  PTP.  The 
operator  would  depress  SCREEN  3  to  activate  the  Sensor  Plan 
format  on  MPD  3 . 

Figure  7  depicts  the  Sensor  Plan  format.  The  top  left  of 
the  format  presents  SA  information.  The  aircraft  is  shown  to  be 
enroute  to  waypoint  (WPT)  7  which  is  6.6  nm  distant.  The 
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Figure  7.  SFFD  Sensor  Plan  Format 
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time-to-go  until  the  next  preplanned  image  acquisition  event  is 
shown  as  is  the  IRIG  (Interrange  Instrumentation  Group)  time 
(H:MM:SS).  The  top  right  portion  of  the  format  is  a  graphic 
presenting  the  current  aircraft  attitude.  It  contains  a  pitch 
ladder  which  also  rotates  to  provide  a  roll  cue  (similar  to  a 
conventional  attitude  display  indicator  instrument)  and  a  heading 
digital  readout.  The  remainder  of  the  format  presents  the  Sensor 
Plan  data.  There  are  13  planned  image  events  in  this  leg,  with 
each  event  shown  on  a  row  in  the  Plan.  The  first  number  in  each 
row  (1,  2,  3,  ...)  identifies  the  event.  The  number  below  the  ID 

number  is  an  SMS-assigned  "confidence”  or  "quality"  value.  In 
the  Figure,  each  of  these  confidences  has  been  (arbitrarily) 
assigned  a  value  of  50.  The  value  in  the  Sensor  Plan  is  based 
solely  on  confidence  in  the  a  priori  intelligence  that  supported 
the  area  limitation  data  base  construction  of  mission  planning. 

A  set  of  four  boxed  letters  (A,  M,  F,  and  L)  appears  in  each  row. 
They  represent  independent  information  sources  and  if  a  box  is 
filled  in,  it  tells  the  operator  that  a  sensor  event  is  planned. 
(During  the  course  of  the  mission,  the  plan  will  be  updated  in 
real-time,  as  imaging  events  actually  occur.)  The  "A"  refers  to 
the  area  limitation  analysis  and  is  always  filled  in  since  that 
is  how  preplanned  imaging  events  are  nominated.  The  "M"  refers 
to  the  MMWR,  the  "F"  to  the  FLIR,  and  the  "L"  to  the  LLS.  For 
HVP  1,  a  single  FLIR  image  is  planned  (the  "F"  box  is  filled  in), 
for  HVP  2  both  FLIR  and  LLS  images  are  planned,  and  for  HVP  3 
only  a  LLS  image  has  been  scheduled. 
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IP-to-End  Search:  During  this  leg  of  the  mission  phase,  the 
operator  again  performs  three  types  of  tasks.  He  monitors  the 
execution  of  the  sensor  plan.  The  primary  information  source  is 
the  SA  display  format  on  MPD  3.  Figure  8  depicts  this  format. 

The  top  portion  of  the  SA  format  contains  two  graphic  counters, 
one  showing  the  number  of  image  buffers  available  for  storing 
FLIP  images  (denoted  by  the  “F")  and  the  other  for  LLS  images. 

It  also  presents  time  and  distance  information  (similar  to  the 
Sensor  Plan  format) .  The  main  area  of  the  format  shows  the 
current  leg  of  the  mission.  The  aircraft  track  is  shown  as  are 
the  preplanned  sensor  points.  Points  co-located  with  area 
limitation  points  are  shown  by  the  addition  of  a  filled  circle  to 
an  open  circle.  Color  codes  are  used  to  highlight  the  current 
(next)  FLIP  and  LLS  imaging  events  and  to  indicate  which  HVPs 
will  be  imaged  by  a  single  sensor  or  by  both  sensors. 
Exclusion/"keep  out"  may  also  be  indicated  by  graphics.  A 
distance  scale  is  provided  for  operator  reference.  The  lower 
portion  of  the  format  presents  aircraft  location,  heading, 
altitude,  and  ground  speed  (GS)  information,  as  well  as  the 
status  of  the  MMWP  (ON/OFF) ,  an  attitude  and  heading  reference, 
and  sensor  imaging  event  infomation. 

The  operator  performs  occupancy  checks  of  the  HVPs  as  they 
are  imaged.  He  uses  the  FLIP  and  LLS  sensor  image  format  screens 
on  MPDs  1  and  2  to  support  this.  These  format  screens  are 
depicted  in  Figure  9  and  their  associated  PTP  menus  are  shown  in 
Figure  10.  The  cursor  box  shown  on  the  FLIP  FPZ  format  indicates 
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Figure  8.  Situational  Awareness  Display  Format 


Figure  9.  FLIR  and  LLS  Freeze  Frame  Image  Display  Screens 
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Figure  10.  PTP  Controls  for  FLIR  and  Laser  Line 
Scan  Freeze  Frame  Control 
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either  the  location  of  the  HVP  (if  intelligence-derived)  or  the 
location  of  the  MMWR  detection  (if  real-time  sensed)  within  the 
displayed  image.  The  cursor  appears  as  a  graphic  overlay  to  the 
sensor  image.  It  is  under  manual  control  and  its  location  on  the 
MPD  is  governed  by  the  knurled  knob  located  on  top  of  the  TH. 

The  operator  uses  the  FLIR  and  LLS  freeze  frame  PTP  screen 
formats  to  execute  the  real-time  occupancy  checks.  As  images 
appear  on  the  FLIR  and  LLS  FRZ  MPD  screens  (Figure  9) ,  the 
operator  uses  the  ISA,  MAYBEA,  and  NOTA  TARGET  buttons  on  the 
appropriate  PTP  (Figure  10)  to  input  his  declaration.  (This 
action  also  assigns  a  confidence  value  to  the  imaged  target.  A 
NOTA  TARGET  entry  has  a  value  of  0,  a  MAYBEA  TARGET  entry  has  a 
value  of  50,  and  an  ISA  TARGET  entry  has  a  value  of  100.  These 
values  are  entered  in  the  Target  Data  MPD  format  screen, 
described  below.)  Additional  information  is  presented  to  the 
operator  regarding  the  generation  of  the  image.  The  "A"  portion 
of  the  A-M-F-L  box  is  filled  in  if  the  image  is  of  a  preplanned 
HVP  and  the  "M”  is  filled  in  if  the  image  is  of  a  real-time 
MMWR/ATC  target  detection.  These  data  elements  may  help  the 
operator  reach  his  decision. 

Having  made  his  (initial)  declaration,  the  operator  can 
depress  the  NEXT  IMAGE  button  on  the  appropriate  PTP  to  call-up 
an  image  currently  stored  in  buffer.  The  SFFD  system  concept  has 
the  capability  to  store  eight  LLS  FRZ  images  and  16  FLIR  FRZ 
images.  ISA  and  MAYBEA  TARGET  declaration  images  are  returned  to 
image  store  buffers.  Images  for  which  the  operator's  declaration 
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is  NOTA  TARGET  are  discarded  from  the  SFFD  system.  Buffer 
counters  are  included  in  the  top  portion  of  the  SA  Awareness 
display  format  (Figure  7). 

To  some  degree,  part  of  the  operator's  task  during  this 
phase  of  the  mission  is  to  manage  the  image  store  queue.  He 
should  be  aware  that  possible  new  images  might  be  lost  if  there 
is  no  place  for  them  in  image  buffer.  The  image  store  buffers 
are  depicted  on  the  FLIR  and  LLS  FRZ  MDP  formats.  These  counters 
are  updated  in  real-time  as  new  images  are  commanded  and 
acquired.  They  are  decremented  in  real-time  as  the  operator 
makes  NOTA  TARGET  assignments.  The  SA  and  Sensor  Plan  formats 
depict  how  many  more  images  are  scheduled.  Of  course,  the 
MMWR/ATC  may  generate  additional  imaging  events  in  real-time.  If 
the  image  store  buffer  is  filled,  no  new  images  will  be  acquired. 
Knowing  that  some  target  acquisition  opportunities  might  be  lost 
places  some  additional  time  stress  on  the  operator. 

The  third  task  performed  by  the  operator  during  this  mission 
segment  is  to  update  target  locations.  This  is  accomplished  by 
using  the  cursor  control  knob  on  the  TH  to  refine  the  target 
location  cursor  graphic  on  the  MPD.  The  cursor  position  is 
adjusted  so  that  it  is  superimposed  directly  over  the  image  of  a 
target  vehicle.  The  trigger  switch  on  the  TH  is  used  to  capture 
the  refined  target  location  (cursor  position)  and  to  transfer  it 
to  a  stores  management  subsystem  for  possible  use  as  a  weapon 
aimpoint . 

End  Search-to-End  Mission  Phase  During  this  portion  of  the 
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scenario,  the  operator  is  concerned  with  completing  any  occupancy 
checks  (for  images  which  are  still  in  buffer  but  which  have  not 
yet  been  reviewed)  and  with  confirming  weapon  assignments. 

During  this  (latter)  task,  the  operator  interacts  with  the  SFFD 
Target  Data  Format  display  screen  (Figure  11) .  This  format  is 
somewhat  similar  to  the  Sensor  Plan  Format  screen  (Figure  6)  with 
respect  to  mission  and  aircraft  attitude  information.  There  are, 
however,  two  major  differences.  The  listing  of  available  images 
now  contains  those  image  identifications  for  which  a  sensor  was 
actually  employed  and  for  which  the  operator  assigned  an  ISA  or  a 
MAYBEA  TARGET  confirmation  response.  (The  probability  or 
confidence  values  shown  were  adjusted  based  on  the  operator's 
prior  review.  An  ISA  TARGET  response  raised  the  value  to  100.) 
The  second  new  feature  of  the  Target  Data  Format  screen  concerns 
weapon  assignments.  The  SFFD  operational  concept  assumed  that 
six  standoff  missiles  were  available  for  target  attack.  A  Weapon 
Status  indicator,  located  at  the  top  of  this  screen,  depicts  how 
many  weapons  have  been  automatically  assigned  against  confirmed 
targets.  This  assignment  is  made  on  the  basis  of  the  adjusted 
confidence  values.  The  six  targets  currently  having  the  highest 
associated  confidences  are  paired  with  weapons. 

The  operator  tailors  the  SFFD  01  (using  the  PTP  4  Select 
Format  and  Select  Screen  commands)  to  activate  the  Target  Data 
Format  screen  on  MPD  3  (bottom) .  An  associated  TARGET  DATA  menu 
of  control  functions  appears  on  PTP  3.  He  then  employs  the  TH 
controls  to  select  an  image  for  review.  He  does  this  by  using 
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WPT:  7 


DIST:  6.6 


TTT  00:00 


IRIG  00:00:00 


00  089  30 


Figure  11.  SFFD  Target  Data  Format 
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the  knurled  knob  on  the  top  of  the  TH  to  slew  a  selection  cursor 
to  the  Target  Data  row  which  presents  the  information 
corresponding  to  the  image  to  be  selected.  Depressing  the  TH 
trigger  switch  activates  of  "hooks”  the  desired  imagery.  (Again, 
LLS  FRZ  imagery  appears  on  MPD  1  and  FLIR  FRZ  images  are 
presented  on  MPD  2 . )  Once  the  target  imagery  has  appeared  for 
his  review,  he  performs  image  study  and  target  location  updating 
(as  above) . 

The  operator  can  interact  with  the  automated  weapon 
assignments  by  using  the  WEAPON  OK  and  WEAPON  NOT  OK  menu  items 
on  the  Target  Data  format  of  PTP  3.  If  WEAPON  OK  is  invoked,  no 
change  occurs.  If,  however,  the  operator  invokes  the  WEAPON  NOT 
OK  control  for  an  assigned  weapon,  this  has  the  effect  of 
deselecting  that  weapon.  Two  changes  to  the  Target  Data  Format 
data  occur  on  MPD  3.  The  weapon  identification  number,  located 
in  the  right  hand  column  of  the  target  data  row,  disappears  and  a 
"free"  weapon  appears  in  the  Weapon  Status  area. 

The  operator  can  also  employ  these  control  functions  to 
nominate  new  targets  for  attack.  This  is  essentially  a  manual 
override  capability  within  the  automated  SFFD  system  concept. 

The  operator  uses  the  TH  to  hook  a  target  image  for  which  no 
weapon  has  been  assigned.  Again,  imagery  appears  on  the  MPDs  and 
image  review  is  accomplished.  A  WEAPON  NOT  OK  input  (PTP  3)  will 
leave  the  weapon  aassignment  situation  unchanged.  A  WEAPON  OK 
input  will,  command  the  assignment  of  a  free  weapon  against  the 
imaged  target.  This  is  equivalent  to  changing  a  MAYBEA  TARGET 
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declaration  to  an  ISA  TARGET  declaration  during  image  review. 

Since  more  images  than  can  appear  on  one  page  of  the  Target 
Data  Format  screen  could  have  been  retained  in  long  term  buffer 
storage,  a  PAGE  UP/PAGE  DOWN  function  is  provided.  These 
controls  are  included  in  the  TARGET  DATA  menu  on  PTP  3. 
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Section  IV 


SUBJECT  MATTER  EXPERTS  (SMEs) 


SMES 

Seven  Strategic  Air  Command  (SAC)  rated  personnel  supported 
the  in-flight  demonstration  portion  of  the  SEED  project.  Three 
were  assigned  to  SAC  Headquarters,  Offutt  Air  Eorce  Base, 
Nebraska,  and  the  remaining  four  were  assigned  to  operational  SAC 
units.  They  were  all  trained  and  experienced  radar  navigators. 
Six  of  the  seven  participated  in  the  project  as  operators  while 
the  seventh  served  as  a  "lead  operator"  during  ground  school  and 
as  an  in-flight  observer  during  all  demonstration  flights.  They 
ranged  in  bomber  flight  experience  from  2100  to  3800  hours,  with 
a  mean  of  2571  hours.  Strategic  aircraft  in  which  this 
experience  was  obtained  included  the  B-IB,  EB-lllA,  B-52H,  B-2 
(avionics  testb-'d)  ,  and  the  SR-71.  Six  had  previous  experience 
using  ELIR  for  navigation  or  targeting  (all  with  the  Electro- 
optical  viewing  system  in  the  B-52) .  This  experience  ranged  from 
1800  hours  to  2250  hours  (with  a  mean  of  1993  hours) .  Six  of  the 
seven  had  previous  experience  in  laboratory  studies  directed  at 
evaluating  the  utility  of  advanced  avionics  capabilities  for 
countering  critical  mobile  targets  (i.  e.,  mobile  missile 
batteries) .  The  laboratory  demonstrations  included  the  concept 
of  a  mission  management  system. 

Highly  experienced  SMEs  were  desired  to  assure  that  they 
would  readily  understand  the  avionics  concepts  and  associated 
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tactics  embodied  in  the  SFFD  project.  Broad  as  well  as  specific 
experience  is  required  to  accomplish  the  orientation  to  future 
concepts  with  minimum  training  time  and  support  (Kuperman,  1984) . 
The  use  of  a  "lead  operator"  was  also  intended  to  facilitate  SME 
orientation.  His  role  was  to  serve  as  an  instructor  during 
ground  school  and  refresher  training.  It  was  hoped  that  by 
receiving  training  from  "one  of  their  own,"  opportunities  for 
ambiguity  or  confusion  on  the  part  of  the  SMEs  would  be  reduced 
to  a  minimum. 

GROUND  SCHOOL 

Ground  school  was  held  at  the  Boeing  facilities,  Seattle, 
Washington,  approximately  90  days  prior  to  the  actual  SFFD 
flights.  It  included  a  combination  of  classroom  and  crew  station 
operations  practice.  The  classroom  portion  covered  requirements 
for  finding  and  attacking  critical  mobile  targets,  the  SFFD 
avionics  concepts,  and  mission  tactics.  During  crew  station 
familiarization  and  practice,  the  actual  SFFD  flight  equipment 
was  employed  in  a  laboratory  (rather  than  a  flight)  environment. 
SMEs  were  trained  in  two  groups,  one  week  apart,  to  assure 
adequate  "hands  on"  experience.  FLIR  and  LLS  imagery,  obtained 
during  prior  test  flights,  was  employed  in  training  to  support 
content  validity. 

Classroom  training  topics  included  an  overview  of  the  SFFD 
project  objectives,  an  overview  of  the  purpose  and  functioning  of 
the  SMS,  orientation  to  the  SFFD  sensor  complement,  description 
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of  the  demonstration  aircraft,  description  of  the  SFFD  01, 
discussion  of  the  workload  metrics  to  be  employed,  description  of 
the  flight  profiles,  and  tours  of  the  SFFD  development  facility. 
Two  days  of  01  practice  were  provided. 


REFERESHER  TRAINING 

Pairs  of  SMEs  participated  in  SFFD  flight  demonstrations 
during  three,  one  week  periods.  The  first  day  was  devoted  to 
refresher  training.  The  objectives  of  the  SFFD  project  were 
reviewed.  Mission  materials  and  plans  were  studied.  The  SMEs 
were  allowed  to  practice  using  the  01  actually  installed  in  the 
testbed  aircraft.  Again,  imagery  from  prior  (non-SFFD)  flights 
was  used  to  support  SME  training. 
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Section  V 


MEASURES 


BASELINE 

Three  measures  of  performance  (MOPs)  were  of  particular 
interest  with  regard  to  the  demonstration  and  evaluation  of  the 
SEED  01: 

1.  The  workload  experienced  by  the  aircrew  in 
conducting  the  target  search,  confirmation,  and  weapon  assignment 
tasks, 

2.  The  ability  of  the  aircrew  (performance)  to 
actually  perform  these  tasks,  given  the  SEED  01,  and 

3.  The  aircrew's  assessment  of  the  SEED  01  itself. 

Workload  was  measured  by  the  Subjective  Workload  Assessment 

Technique  (SWAT) .  In  SWAT,  workload  is  defined  as  a  three 
dimensional  combination  of  time  stress  (T) ,  mental  effort  (E) , 
and  psychological  stress  (S) .  In  practice,  SWAT  is  administered 
by  means  of  a  two-step  process:  individual  scale  development  and 
event  scoring.  During  individual  scale  development,  each  SME 
rank  orders  a  set  of  27  unique  combinations  of  T,  E,  S 
descriptors.  Conjoint  analysis,  a  mathematical  process,  is 
employed  to  generate  an  unidimensional  SWAT  workload  scale  over 
the  interval  0  to  100.  During  event  scoring,  the  SME  is  prompted 
to  provide  a  SWAT  T,E,S  triplet  at  the  completion  of  each  defined 
task.  The  SWAT  triplet  is  converted,  by  table  look-up  using  the 
individual  SME  scale,  into  SWAT  workload  values. 
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Performance  for  the  SFFD  project,  is  expressed  in  terms  of 
the  percentage  of  targets  correctly  confirmed  (hit  rate,  HR)  and 
the  percentage  of  non-target  images  incorrectly  confirmed  by  the 
aircrew  (false  alarm  rate,  FAR) .  These  MOPs  are  calculated  based 
on  the  FLIR  and  LLS  images  cued  to  the  operator  as  possible 
target  detections  (by  the  MMWR)  or  as  preplanned  HVP  checks. 

The  third  MOP  deals  with  the  assessment  of  the  SFFD  01 
concept.  The  instrument  employed  to  quantify  this  MOP  is  a  set 
of  rating  scales.  First,  the  element  of  the  SFFD  01  to  be 
assessed  is  defined.  Next,  the  SME  is  required  to  mark  a  seven 
point  rating  scale  to  indicate  how  the  SSFD  01  relates  to  the 
design  element.  (The  first,  fourth,  and  seventh  demarcations  on 
the  scale  have  semantic  anchors  associated  with  them.  The  left- 
hand  end  of  the  scale  is  always  anchored  at  a  negative  response 
(e.  g. ,  greatly  detracts  from  effectiveness],  the  middle  position 
is  always  neutral  [e.  g. ,  neither  enhances  nor  detracts  from 
effectiveness],  while  the  right-hand  anchor  is  always  associated 
with  a  positive  response  [e.  g.,.  greatly  enhances 

effectiveness].)  Lastly,  the  SME  is  solicited  to  provide  written 
comments  to  substantiate  or  expand  on  his  rating  scale  response. 

The  rating  scales  dealt  with  the  following  SFFD  01  areas: 

1.  Support  of  SA  during  the  Pre-IP  phase 

2.  Effectiveness  of  the  Sensor  Plan  Format  screen  during 
the  Pre-IP  phase 

3.  Effectiveness  of  the  SA  Format  screen  during  the  target 
search  phase 
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4.  Effectiveness  of  the  LLS  FRZ  and  FLIR  FRZ  format  screens 
during  the  target  search  phase 

5.  Effectiveness  and  unambiguity  of  color  coding  on  the  SA 
Format  screen 

6.  Effectiveness  of  the  Target  Data  Format  screen  during 
the  weapon  assignment  phase 

7.  Utility  of  the  TH  controls  during  the  confirm  weapons 
phase 

8.  Overall  ease  of  use  and  crew  acceptance  of  the  SFFD  01 
controls  and  displays  (MPDs,  PTPs,  TH) 

9.  Availability  of  information  during  the  confirm  weapons 
phase 

10.  Overall  adequacy  of  the  size  and  arrangement  of  the  MPDs 
and  PTPs 

11.  Preference  for  freeze  frame  image  presentation  versus 
"live”  sensor  video 

12.  Expected  acceptance  of  the  SFFD  SMS  concept  by  the 
operational  community. 

SFFD  01  ENHANCEMENT  CONCEPTS 

The  SFFD  01  (as  described  above)  was  actually  employed  by 
the  SMEs  during  the  course  of  the  flight  demonstration  activity. 
Several  additional  features  of  the  01,  which  might  be 
incorporated  in  a  future  SFFD  01  design,  were  demonstrated  or 
described  to  the  SMEs  at  the  conclusion  of  the  in-flight  portion 
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of  the  project.  These  were: 

1.  A  counter  which  tallied  the  number  of  times  that  the 
SME  had  viewed  a  given  image  (including  both  the  target 
search  and  weapon  assignment  phases  of  the  mission) . 
(This  feature  had  been  suggested  independently  by  two  of 
the  SMEs  during  ground  school) . 

2.  The  addition  of  eight  more  image  buffers  for  the  storage 
of  LLS  FRZ  images.  (The  01,  as  flown,  could  store  eight 
LLS  FRZ  images  and  16  FLIR  FRZ  images  for  subsequent  SME 
review) . 

3.  Two  types  of  image  underlay  on  the  SA  Display  Format 
screen  were  demonstrated.  The  first  was  digital  terrain 
elevation  data  (DTED)  produced  by  the  Defense  Mapping 
Agency  (DMA) .  The  DTED  was  color  coded  with  respect  to 
the  terrain  elevation.  The  second  underlay  approach  was 
based  on  digitizing  color  aerial  photographs  of  the  Ft. 
Lewis  area. 

OPERATIONAL  UTILITY  ASSESSMENT 

The  Ft.  Lewis  range  constrained  the  duration  of  SFFD  flight 
scenarios.  In  general,  a  target  search  pass  was  flown  in  less 
than  two  minutes.  During  the  post-mission  debrief,  the  SMEs  were 
shown  the  mission  timeline  depicted  in  Figure  12.  In  this 
scenario,  target  search  (as  depicted  by  the  rectangular  box 
overlaying  the  aircraft  track)  has  a  duration  of  five  minutes. 

The  SMEs  were  briefed  on  two  scenarios:  a) that  there  was  half 
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Figure  12.  Notional  Operational  SFFD  Scenarios 
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the  target  density  in  the  search  area  and  b)that  there  was  the 
same  target  density  in  the  search  as  they  had  experienced  during 
the  SFFD  flights.  Assuming  that  there  were  four  actual  targets 
present  in  the  search  area,  they  were  asked  to  rate  their 
expectation  that  the  SFFD  system  would  have  allowed  them  to  fire 
at  least  two  missiles  against  targets  for  which  they  were  at 
least  80  percent  confident.  They  were  also  asked  to  projectively 
provide  a  SWAT  rating  (Pro-SWAT)  for  the  workload  that  they  would 
expect  to  experience  under  these  more  operationally  realistic 
conditions . 

OPTIMIZING  THE  SFFD  01 

Recognizing  that  the  SFFD  OI  was  only  one  of  many  possible 
crew  system  interface  concepts  for  use  with  SFFD  avionics 
capabilities,  the  SMEs  were  asked  to  comment  on  four  of  its  major 
design  attributes:  display  placement,  display  size,  symbology, 
and  format  content.  They  were  also  solicited  to  provide  written 
comments  (both  individually  and  in  the  form  of  a  group  consensus) 
as  to  how  they  would  redesign  the  SFFD  OI  to  make  it  better 
suited  to  support  them  in  the  search,  occupancy  check,  and  weapon 
confirmation  tasks. 
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Section  VI 


RESULTS 


WORKLOAD 

Workload  (SWAT)  ratings  were  obtained  from  the  seven  SMEs 
immediately  after  the  completion  of  two  distinct  SEED  tasks:  Pre- 
IP  situational  awareness  and  target  confirmation/weapon 
assignment.  SWAT  responses  from  93  of  the  SEED  passes  were 
selected.  The  selection  was  based  on  passes  in  which  all  SEED 
subsystems  were  in  proper  operation.  (This  was  done  based  on  the 
flight  logs.)  The  mean  SWAT  workload  values  were  computed  to  be; 

Pre-IP  Target  Confirmation/ 

Weapon  Assignment 

Mean  SWAT  4.6  26.3 

(N  =  93) 

Reid  and  Colle  (1988)  provide  guidance  on  interpreting  SWAT 
scores.  They  suggest  that  a  value  in  the  range  of  30  to  50  is 
indicative  of  a  potential  workload  saturation  condition. 

PEREORMANCE 

Although  the  SEED  project  was  intended  to  be  a  demonstration 
of  an  avionics  concept  and  not  an  operational  assessment  of  the 
capabilities  of  that  concept,  an  indication  of  the  effectiveness 
of  system  can  provide  context  within  which  the  SEED  01  can  be 
assessed  more  realistically.  Woolet  (1992)  presents  a 
preliminary  analysis  of  SEED  system  performance.  Of  the  864 
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images  actually  acquired,  186  of  them  contained  actual  targets 
(not  including  empty  nets  or  vehicles  totally  masked  t-  terrain 
or  vegetation) .  The  SMEs  were  able  to  correctly  confirm  117  of 
these,  resulting  in  a  HR  of  almost  63  percent  (117/186).  In 
addition,  they  incorrectly  declared  60  non-target  images  to 
contain  targets,  resulting  in  a  FAR  of  eight  percent  (60/ [864  - 
117])  . 

SFFD  01  ASSESSMENT 

Figure  13  A  and  B  presents  the  mean  of  the  SFFD  01 
assessment  ratings  obtained  from  the  seven  SMEs  in  bar  graph 
form.  In  the  rating  scale  questionnaire  and  on  the  Figure,  a 
value  of  four  represents  a  neutral  response  on  the  part  of  the 
SME.  None  of  the  SFFD  01  attributes  explored  by  the  rating  scale 
questionnaire  received  a  mean  rating  of  less  than  4.6,  indicating 
that  the  SMSs  felt  at  least  mildly  positive  about  that  SFFD  01 
feature . 

The  three  questions  which  produced  the  most  strongly 
positive  mean  SME  ratings  (i.  e.,  strongest  SME  agreement)  were 
as  follows: 

1.  FRZ  LLS  and  FRZ  FLIR  are  a  strong  improvement  over 
"live”  video  imagery  for  SFFD  target  acquisition 
tasks  (mean  SME  rating  =  6.0) 

2.  The  complexity  of  the  SFFD  01  controls  and  displays 
was  at  a  level  acceptable  to  the  operational  user 
community  (mean  SME  rating  =  5.9) 
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SFFD  01 
ATTRIBUTE 

SUPPORTS  SA  IN 
PRE-IP  PHASE 


SENSOR  PLAN  FORMAT 
EFFECTIVE  IN  PRE-IP 

SA  FORMAT  EFFECTIVE 
IN  TARGET  SEARCH 


LLS  &  FUR  FRZ  FORMATS 
EFFECTIVE  IN  SEARCH  AND 
WEAPON  CONFIRMATION 

COLOR  ON  SA  FORMAT 
EFFECTIVE  AND  NOT 
CONFUSING 

TARGET  DATA  FORMAT 
EFFECTIVE  IN  CONFIRM 
WEAPON  ASSIGNMENTS 

STRONGLY 

NEGATIVE 


RATING 

4 


NEUTRAL 


Figure  13A.  Results  of  Rating  Scale  for  Baseline  SFFD 
Operator  Interface  Design  Attributes 


STRONGLY 
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SFFD  OI 
ATTRIBUTE 


OPERATION  OF  TH  IN 
CONFIRM  WEAPONS 


ACCEPTABLE  EASE  OF  USE 
OF  CONTROLS  &  DISPLAYS 

READY  AVAILABILITY  OF 
CONFIRM  WEAPONS  INFORMATION 


SIZE/ARRANGEMENT  OF  MPDs 
AND  PTPs  ACCEPTABLE 


ACCEPTANCE  OF  SMS  BY 
USER  COMMUNITY 


FREEZE  FRAME  IMAGES 
IMPROVEMENT  OVER  “LIVE"  IMAGES 


STRONGLY 

NEGATIVE 


NEUTRAL 


STRONGLY 

POSITIVE 


Figure  13B.  Results  of  Rating  Scale  for  Baseline  SFFD 
Operator  Interface  Design  Attributes 
(Continued) 
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3 .  Acceptance  of  the  demonstrated  SMS  concept  by  the 
operational  user  community  (mean  SME  rating  =  5.7) 


OPERATIONAL  UTILITY  ASSESSMENT 

For  the  two  target  density  cases  (half  and  equal) ,  the  SMEs 
strongly  felt  that  they  would  be  at  least  80  percent  confident  in 
being  able  to  search,  confirm,  and  assign  weapons  against  at 
least  two  of  four  possible  targets  (mean  SME  ratings  =  6.0  and 
5.7,  respectively).  In  the  case  of  half  the  target  density  but 
extended  search  duration,  the  SMEs  expected  to  experience 
approximately  the  same  workload  (mean  SME  Pro-SWAT  rating  =  25.2 
[versus  mean  SME  SWAT  rating  =  26.3])  as  encountered  during  the 
SFFD  flights.  For  the  case  of  the  same  target  density  and 
extended  duration,  they  expected  to  encounter  a  significantly  (p 
<  0.01)  elevated  level  of  workload  (mean  SME  Pro-SWAT  rating  = 
41.3. 

SFFD  01  ENHANCEMENTS 

The  SMEs  saw  little  or  no  value  with  regard  to  adding  a 
counter  to  tally  the  number  of  times  an  image  had  been  viewed 
(mean  SME  rating  =  3.8).  They  did  see  positive  benefit  to  adding 
an  additional  eight  image  storage  buffers  for  LLS  FRZ  imagery 
(mean  SME  rating  =  5.5).  They  verbally  expressed  a  strong 
preference  for  the  digitized  color  photography  as  an  underlay  on 
the  SA  Display  Format  screen  (verbal  comments  only) . 
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SFFD  01  OPTIMIZATION 

Several  general  observations  appeared  in  the  individual  and 
consensus  responses  to  SFFD  01  optimization.  The  SMEs  desired  to 
have  the  01  controls  and  displays  more  closely  grouped  together 
and  integrated  within  a  lower  console.  (They  had  to  look  up  to 
employ  MPD  1  and  2  during  the  flight  demonstrations.)  They 
generally  desired  to  have  all  PTP  functions  consolidated  into  an 
integrated  keyboard,  located  to  the  left  side  of  the  01  console. 
This  was  expected  to  facilitate  left  hand  operation  of  all  task 
functions  requiring  keyboard  entry  while  leaving  the  right  hand 
free  for  TH  functions.  (During  the  demonstration,  SMEs  had  to 
remove  their  right  hand  from  the  TH  to  control  PTPs  2  and  4 
[Figure  5]).  They  unanimously  found  the  size  of  the  SFFD  01 
displays  (approximately  six  by  eight  inches,  vertical  and 
horizontal  dimensions)  to  be  acceptable.  They  felt  that  the 
capability  to  declutter  the  FLIP  FRZ  was  desirable.  (They  could 
declutter  overlayed  symbology  from  the  FLIP  FPZ  format.)  They 
also  strongly  desired  to  have  an  image  "zoom"  or  magnification 
capability  integrated  into  the  FLIP  and  LLS  FRZ  image  display 
formats  so  that  they  could  manually  command  an  enlargement  of  the 
image  for  suspected  targets. 
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Section  VII 


CONCLUSIONS 


WORKLOAD 

The  workload  levels  reported  by  the  seven  SMEs  who 
participated  in  the  SEED  project  reflected  adequate  support  by 
the  SEED  avionics  integration  concept  (which  included  both  an 
automated  SMS  and  tailored  01) .  Workload  associated  with  Pre-IP 
situational  awareness  activities  was  comparable  to  that 
frequently  found  associated  with  the  crew  monitoring  the 
automated  functioning  of  a  subsystem  (e.  g.,  an  automated  weapon 
release  event) .  The  mean  SWAT  score  associated  with  the  target 
acquisition  tasks,  although  much  higher,  was  not  in  the  range 
which  would  indicate  task  saturation.  Based  on  these  results, 
the  workload  found  to  be  associated  with  the  SEED  concept  can  be 
qualitatively  described  as  being  in  the  low-to-medium  range. 

PEREORMANCE 

Although  not  of  primary  concern  in  the  context  in  an 
avionics  concept  demonstration,  the  values  obtained  for  the  HR 
(0.63)  and  EAR  (0.08)  MOPS  are  very  supportive  of  the  further 
development  of  the  SEED  concept.  These  MOPs  can  be  expected  to 
show  improvement  if  SPED  avionics  upgrades  are  made.  A  first  or 
second  generation  PLIR  can  be  expected  to  provide  greatly 
enhanced  image  quality  over  the  commercial  ELIR  used  in  the 
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demonstration.  ATC/ATR  technology  could  be  applied  to  both  the 
LLS  and  FLIR  imaging  systems  in  expectation  of  reducing  the 
number  of  non-target  images  presented  to  the  operator.  An  image 
zoom  capability  could  help  the  operator  reduce  both  the  FAR  (and 
the  number  of  missed  targets) .  A  “tighter  and  lower" 
rearrangement  of  the  01  might  reduce  operator  workload. 


SFFD  01 

The  general  findings  of  the  SMEs  was  that  the  SFFD  01  was 
both  effective  and  highly  acceptable  by  the  user  community. 
Several  suggestions  (above)  for  01  improvement  were  made  which 
might  further  enhance  the  effectiveness  of  the  01. 


OPERATIONAL  UTILITY 

Although  the  SMEs  generally  expressed  confidence  in  being 
able  to  effectively  employ  the  SFFD  avionics  capabilities 
(including  the  01)  under  more  realistic  operational  scenario 
conditions,  the  relatively  high  Pro-SWAT  value  produced  for  the 
high  target  density,  extended  duration  mission  is  of  some 
concern.  If  the  SFFD  avionics  concept  is  further  developed, 
attention  should  be  paid  to  reducing  crew  workload  either  through 
additional  avionics  capabilities  or  a  higher  level  of  crew 
training. 


ENHANCEMENTS 

The  image  review  counter  was  judged  to  be  of  little  value 
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and  probably  should  be  further  pursued  in  any  future  development. 
The  addition  of  LLS  image  buffers  and  the  use  of  digitized  color 
aerial  photograplu'  as  an  underlay  on  the  SA  Display  format  screen 
were  both  supported  by  the  SMEs  and  should  be  seriously 
considered  for  incorporation. 

OPTIMIZATION 

Although  the  SEED  01  was  generally  highly  acceptable  to  the 
SMEs,  several  suggestions  were  elicited  which  might  further 
improve  the  crew  system  design.  The  "tighter  and  lower" 
arrangement  of  the  SEED  01  controls  and  displays  and  the  addition 
of  an  image  zoom  capability  are  suggested  for  any  further 
development . 
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